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Abstract 


In multivariate statistics, the linear relationship among random variables has been fully explored 
in the past. This paper looks into the dependence of one group of random variables on another 
group of random variables using (conditional) entropy. A new measure, called the K-dependence 
coefficient or dependence coefficient, is defined using (conditional) entropy. This paper shows that 
the K-dependence coefficient is a measure of the degree of dependence of one group of random 
variables on another group of random variables. The dependence measured by the K-dependence 
coefficient includes both linear and nonlinear dependencies between two groups of random 
variables. Therefore, the K-dependence coefficient measures the total degree of dependence and 
not just the linear component of the dependence between the two groups of random variables. 
Furthermore, the concept of the K-dependence coefficient is extended by defining the partial 
K-dependence coefficient and the semipartial K-dependence coefficient. Properties of partial and 
semipartial K-dependence coefficients are also explored. 


Key words: Entropy, conditional entropy, K-dependence coefficient, partial K-dependence 
coefficient, semipartial K-dependence coefficient 
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1. Introduction 


Entropy is a measure of uncertainty for a random system or a group of random variables. Let 
X = (x\,X 2 , ■ ■ ■, x n ) T and Y = (yi,y 2 ,..., y m ) T be two groups of discrete random variables with 
N and M possible states, respectively, which are denoted by {x\,... ,xn} and {y \,..., vm}- Joint 
probabilities of these possible states are denoted by p(xi, yj) (i = 1 ,... ,N;j = 1,..., M), while 
marginal probabilities are denoted by p{x %) (i = 1,..., N) and p(yj) (j = 1,..., M). It is obvious 
that 

N M 

p^yj ) = 1 

i=l j=1 
N 

£**) = 1 
i =1 

M 

= 1 - 

3 = 1 

Definition 1. 

Entropy in terms of a group of discrete random variables is defined as 

N 

H {X) = - Y.P&) ln (M^))- ( 4 ) 

i= 1 

Consider that 0 < p(xi) < 1 for i = 1,2,..., iV, H(X) in (4) is greater than or equal to zero; 
that is, 


(1) 

( 2 ) 

(3) 


H(X) > 0. 


Thus, entropy can be thought as an extension of the determinant of the covariance matrix of 
X. For the case of a single variable X, entropy is an extension of the variance of X. 


Definition 2. 

Conditional entropy of X given Y is defined by 

N M 

h(x\y) = -YX1 p(xi,yj)ln(p(xi\yj)), (6) 

i=l j =1 
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where p(xi,yj ) is the joint distribution of (x,y), and p(x t \yj) is the conditional probability of X{ 
given y r 

Similar to (5), 


H(X\Y) > 0. (7) 

Conditional entropy H(X\Y) measures the conditional uncertainty of X given Y. In the case 
of N = M = 1, conditional entropy is analogous to the residual variance of a random variable X 
given another random variable Y. It is plausible that conditioning on Y should not increase the 
uncertainty about X. This fact is expressed by the following theorem. 

Theorem 1. 

H(X\Y) < H(X) , with equality if and only if X and Y are independent. (8) 

In (8), equality is equivalent to the independence condition. This implies that entropy H(X) 
and conditional entropy H(X\Y) contain information about the relationship or dependence 
between X and Y. In past decades, (conditional) entropy had been explored as a means for 
measuring the association among random variables. If X and Y are both categorical variables (so 
that the joint frequency distribution for X and Y forms a two-way contingency table), Goodman 
and Kruskal (1954) proposed a number of methods in probabilistic terms to measure the degree 
of association between the two categorical random variables, which are represented by the raws 
and columns, respectively. Thiel (1970) proposed what he called the uncertainty coefficient (U), 
which is defined by the ratio of difference between the entropy of X and the conditional entropy 
of X given Y (information) to the entropy of X. This quantity is also described in Agresti (2002, 
pp. 56-57) and is available in both SAS and SPSS as part of the analysis of two-way contingency 
tables. Haberman (1980, 1982) also discussed the measures of associations and U in the context of 
analyzing multinomial response data. In this paper, a set of measures for the dependence of one 
group of random variables on another group of random variables is defined in terms of entropy 
and conditional entropy. The properties of these measures are explained and studied with the 
mathematical proofs. 
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2. K-Dependence Coefficient 

First, define a new measure, which is called the K-dependence coefficient or dependence 
coefficient, for measuring the degree of dependence between two groups of random variables. 
Then, several properties of the K-dependence coefficient will be explored. 

Definition 3. 


where 


K(X : Y) = «(*)-«(*!?> , 
Hi X) 


H(X) is the entropy of X , and 


(9) 


H(X\Y) is the conditional entropy of X given Y. 

K(X : Y ) in (9) is called the K-dependence coefficient, or dependence coefficient, of X to Y, 
or X dependence on Y. 

In Definition 3, the K-dependence coefficient is defined by the ratio of the difference between 
the entropy of X and the conditional entropy of X given Y, and the entropy of X. The difference 
between the entropy of X and the conditional entropy of X given Y actually measures the degree 
to which X depends on Y, or Y depends on X. With the K-dependence coefficient, the scale of 
this dependence measure is normalized using the entropy H(X). The entropy H(Y ) can also be 
used for this normalization. The interpretations of these two normalizations are different and will 
be discussed later on. 


Property 1. 

The K-dependence coefficient ranges between 0 and 1 when H(X ) > 0; that is, 

0 < K{X : Y) < 1. (10) 

Proof: This is obvious from (5) and Theorem 1. 

Property 1 shows that K(X : Y) is a normalized measurement with the scale between 0 and 
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1. The following theorems look into K(X : Y ) when its value is 0 or 1. 

Theorem 2. 

K(X : Y) = 0 if and only if X and Y are independent f rom, each other. (11) 

Proof: From (9), it is obvious that 

K(X : Y) = 0, if and only if H(X\Y ) = H{X). 

According to Theorem 1, 

H(X\Y) = H(X) if and only if X and Y are independent. 

Therefore, 

K(X : Y) = 0 if and only if X and Y are independent from each other. 


Because the independence of X from Y is equivalent to the independence of Y from X, the 
following property is obvious. 

Property 2. 

K(X :Y) = 0 if and only if K(Y : X) = 0. (12) 

Proof: It is obvious from Theorem 2. 

Theorem 2 looks into the case of K(X : Y) = 0, which is equivalent to independence between 
X and Y. The following theorem looks into the case of K(X : Y) = 1. 

Theorem 3. 

Assume that p(xi) > 0 and p(ijj ) >0 (i = 1, 2,..., TV; j = 1,2,..., M). Then 

K(X :Y) = 1 
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if and only if for each yj, 3 Xi- such that p{X = Xj. |Y = yj) = 1 and p(X = x k \Y = yj) = 0, 
where k 7 ^ ij. 

Proof: First, necessity is obvious because H(X\Y) = 0, from Definition 2, also noting that 
0ln(0) = 0. Therefore, K{X : Y) = 1 by Definition 3. 

Now, assume that K(X : Y) = 1. From Definition 3, 

H(X\Y) = 0; (13) 

that is, 


M N M N 

o = h(x\y) = -Y P(yj)^2p{xi\yj)ln{p(xi\yj) = - EE p(xi,yj)ln(p(xi\yj)). 

j=1 i=l j=1 i=1 

Therefore, 

M N 

EE p(xp yj) ln(p(xi\yj )) = 0. (14) 

j= 1 i=1 

Also, it is noted that 


0 < p(xi, yj) < 1 , and 
0 < p{xi\y.j) < 1. 

Therefore, 

p(xi,yj)ln(p(xi\yj)) < 0 for all i andj. (15) 

Combining (14) and (15), 

p(xi,yj)ln(p(xi\yj)) = 0 for all i andj. (16) 

For each given y), 
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(17) 


N 

=K%) > 0 ,and 

2=1 

p(xi,yj) > 0 for all i. (18) 

Here, p(yj) > 0 is the assumption. From (17), there exists at least one ij, such that 

p{xi j ,Vj)> 0. (19) 

From (16), 

p{xij,yj) Mp(^|%)) = 0. ( 20 ) 

Combining (19) and (20), 

MpO^Ij/j)) =0. (21) 

Therefore, 


Also, 


p(xij\yj) = i- 


( 22 ) 


Combining (22) and (23), 


N 

p^iWj) = i. 

2=1 


(23) 


p(xiWj) = 0 for all i / ij. (24) 

Therefore, (22) and (24) are the proof for sufficiency. 

Theorems 2 and 3 show that K(X : Y ) measures a total dependence or total relationship, 
not only the linear part of this dependence or relationship, between X and Y. More specifically, 
K(X : Y) actually measures the X’s dependence on Y. Generally speaking, this dependence is 
not necessarily symmetric; that is, K(X : Y) may or may not be the same as K(Y : X). This is 
a different situation from linear correlation, where the relationship is always symmetric. Because 
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K{X : Y) measures X’s dependence on Y. while K(Y : X ) measures Y' s dependence on X, 
K(X : Y) has no property of symmetry. 

Next is a proof for a K-dependence coefficient inequality. 

Property 3. 

K(X : Y) < K(X : Y,Z), (25) 

with equality if and only ifX and Z are conditionally independent given Y. 

Proof: Assume X,Y, and Z have N, M, and L states, respectively, which are denoted by 
{xi,....,xjv}, {y\ ,... ,;(/Af}, and {zi,..,,z L }. 

First, 

N M L 

H(X,Z\Y) = - EEE P(xi , Vj,z k ) In (p(xi, z k \yj )) 

i= 1 j= 1 fc=l 
N M L 

= -EEE j/j, 4) ln(p(4|4)p(4|4,4)) 

i=l j=1 fc=l 

N M L N M L 

= -EEE p(4,4,4) ln (p(4l4)) - EEE p(4, yj, 4) ln(p(4|4, 4)) 

i=l j= 1 fc=l i=l j = l fc=l 

= F(z|y) + /f(A|y,z). 

Therefore, 

fl"(X,Z|y) = H(Z\Y) + H(X\Y,Z). (26) 

Second, it has been known that 

ln(x) < x — 1 , with equality if and only if x = 1 . 
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Therefore, for each i, j, and k, 


]nf p(^iWj)p(^k\y j ) ^ < p(xj\yj)p(z k \yj) _ 1 
p(xi,z k \yj ) “ p(xi,z k \yj) 


(27) 


with equality if and only if p(xi, 4 |%) = p(xi\yj)p(z k \yj). 
Multiply (27) by p(xi, 4|%) and sum over i and k to obtain 


N L 

EE p(xi,z k \yj) ln( 

i =1 fc=l 


p(4|%M4|%) 

p(4,4|%) 


N L 

)SEE p(xi,Zk\yj)( 

i=1 k=l 


p(xi\yj)p(zk\yj) 

p(4,4|%) 


i) 


N L 

= E £(p( 4|%>(4|%) - p{xi, 4|%)) = 1-1 = 0, (28) 

i =1 k =1 


with equality if and only if p( 4 , 4 |%) = p( 4 |%)p( 4 |%) for all i,j, and k. 
From (28), 


N L N L 

EE p(4,4|%)in(p(4|%)) + EE p(4,4|%) ln(p(z k \yj)) 

i =1 k =1 i =1 fc=l 

AT L 

sEE p{xi, 4|%) ln(p(xj, 4|%)), 
i =1 fc=l 


(29) 


with equality if and only if p(4,4|%) = p(4|%)p( 4|%) /or all j. 
Multiply (29) by —p(%) and sum over j to obtain 


M N L M N L 

P{xi, 4|%) ln(p(4|%)) ~£p(Vj) EE p(4,4|%)ln(p(4|%)) 

j= 1 i=l k =1 j=l i=l fc=l 

M N L 

p(xi, 4|%) in 4(4,4|%)), 

j=l i=l fc=l 


( 30 ) 


with equality if and only if p(xi, 4 |%) = p( 4 |%)p( 4 |%) for all i,j , and k ; 



that is 


H(X, Z\Y) < H(X\Y) + H(Z\Y), 


(31) 


with equality if and only if p(xi, ZkWj) = p(%i\yj)p(zk\yj) for all i,j, and k. 

From (26) and (31), 

H(X\Y, Z) = H(X, Z\Y) - H(Z\Y ) 

< H(X\Y) + H{Z\Y) - H{Z\Y) = H{X\Y). 

Therefore, 

H(X\Y,Z)<H(X\Y), (32) 

with equality if and only if X and Z are conditionally independent given Y. 

From Definition 3, 

K{X : Y) < K(X : Y, Z), 

with equality if and only if X and Z are conditionally independent given Y. 

Property 3 is also true for the case of multivariables; that is: 

Property 4. 

K(X:YY p ) <K(X:YY p , Y p+1 ), 
with equality if and only if X and Y p+ 1 are conditionally independent given Y \,..., Y p . 

3. Partial and Semipartial K-Dependence Coefficients 

Section 2 defines the K-dependence coefficient, which measures the degree to which a group 
of random variables depends on another group of random variables. This section looks into a 
conditional K-dependence coefficient, which measures the degree to which one group of random 
variables depends on a second group of random variables, given a third group of random variables. 
This conditional K-dependence coefficient is called the partial K-dependence coefficient, or the 
partial dependence coefficient. 
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Let Z = (zi,Z 2 , ■ ■ ■, zi) T be a group of discrete random variables with L possible states, which 
are denoted by {z \,..., zl}- The probability of each state is denoted by p(zk) (k = 1,2,..., L). 

Definition 4. 


where 


K(X : Y\Z) 


H(X\Z) - H(X\Y,Z) 
H(X\Z) 


(33) 


H(X\Z) is the conditional entropy of X given Z, and 

H(X\Y, Z) is the conditional entropy of X given Y and Z. 

K(X : Y\Z) defined in (33) is called the partial K-dependence coefficient , or the partial 
dependence coefficient of X to Y, or X dependence on Y given Z. 


Theorem 4. 

K(X : Y\Z) = 0 if and only if X and Y are conditionally independent given Z. (34) 
Proof: From (32), it is obvious that 

H(X\Z) = H(X\Y,Z) (35) 

if and only if X and Y are conditionally independent given Z. 

Also, from Definition 4, (35) is true if and only if 

K(X : Y\Z) = 0. (36) 

Therefore, (36) is true if and only if X and Y are conditionally independent given Z. 
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Theorem 5. 


Assume that p(xi ) > 0, p('Pj) > 0 and p(z k ) >0 (* = 1,2, ..., N; j = 1,2, ,.., M; k = 

1,2 ,L) Then 

K(X : Y\Z) = 1 

if and only if for each yj and z k , 3 x l;j k , such that p(X = x t/ k \Y = yj,Z = z^) = 1 and 
p{X = xi\Y = yj, Z = z k ) = 0, where l / i jtk . 

Proof: From Definition 4, 


K(X : Y\Z) = 1 (37) 

if and only if 

H{X\Y,Z) = 0. (38) 

Now, define 

$ = (f,Z) T . (39) 

$ is also a vector of random variables with M x L states: 

4>j,k = (yj,z k ) T - (40) 

Then (38) becomes 

H(X |<£) = 0. (41) 

By applying Theorem 3 to $ instead of Y , it can be shown that (41) is true if and only if for 
each (j>j t k , 3 Xi j k such that p(X = x l;i fc |$ = 4>j,k) = 1 and p(X = = (j)j,k) = 0, where l / ij 

This is the same as the statement that for each yj and z k , 3 Xi jk such that 
p(X = x ij k \Y = pj, Z = z k ) = 1 and p(X = x t \Y = pj, Z = z k ) = 0, where l / i jtk . 

Also, from Definition 4, (41) is true if and only if 
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K(X : Y\Z) = 1. 


This proves Theorem 5. 

Similar to Property 3, the partial K-dependence coefficient satisfies the following inequality. 

Property 5. 

K(X-.Yi\Z)<K(X-.Yi,Y 2 \2), (42) 

with equality if and only if X and Y 2 are conditionally independent given Y\ and Z. 

Proof: In the same way as (26) and (31), it can be proved that 

H(X,Y 2 \Y u Z) = H(Y 2 \Y u Z) + H{X\Y u Y 2 ,Z), and (43) 

+ (44) 

with equality if and only ifX and Y 2 are conditionally independent , given Y\ and Z. 

From (43) and (44), 


H{X\\\,Y 2 ,Z) = H{XX-2\YiJ) - H(Y 2 \Y x ,Z) 
< ^|?i, z) + H(Y 2 \Y u Z) - H(Y 2 \Y u Z) = H(X\Y\, Z). 


Therefore, 


H{X\Y U Y 2 ,Z) <H{X\Y U Z), 

with equality if and only if X and Y 2 are conditionally independent , given Yi and Z. 
From Definition 4, 
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K(X:Y\Z) <K{X:%Y 2 \Z), 

with equality if and only if X and Y\ are conditionally independent , given Y\ and Z. 

Definition 5. 


where 


K(X : (Y\Z)) 


H(X\Z) - H(X\Y,Z) 

ml) 


( 45 ) 


H(X\Z) is the conditional entropy of X given Z. and 
H(X\Y, Z) is the conditional entropy of X given Y and Z. 


K(X : (Y\Z)) in (45) is called the semipartial K-dependence coefficient , or the semipartial 
dependence coefficient of X to Y, or X dependence on conditional Y given Z. 

The semipartial K-dependence coefficient differs from the partial K-dependence coefficient by 
a factor of H{X\Z)/H(X); that is, 


Therefore, 


K(X : (Y\Z)) = K(X : Y\Z) x , and 

H(X) 

K(X : (Y\Z)) < K(X : Y\Z). 


(46) 


Theorem 6. 

1 - K(X : Yi,, Y n ) = (1 - K(X : Ki)) x (1 - I<(X : F 2 |Ki)) (47) 

x • • • x (1 — K(X : Y n \Yi ,..., Y n _\)). 
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Proof: 


1-K(X :Y u ...,Y n ) 


H{X\Y l ,...X) 

H(X) 


H(X |fi) x H{X\Y u Y 2 ) x H(X\Y u Y 2 ,Y 3 ) x ___ x 

h(x) x ^(Xin) x f?(x|Fi,y 2 ) x '" x A(x|ii,...,y n _i) 


= (1 - K(X : Pi)) x (1 - Jv(X : K 2 |Ki)) x ■ • ■ x (1 - K(X : Y n \Y U ■ • • X-i))- 


Theorem 6 provides a decomposition of the K-dependence coefficient in terms of the partial 
K-dependence coefficient. The following theorem expresses the partial K-dependence coefficient in 
terms of the K-dependence coefficient. 


Theorem 7. 


Proof: 


1 - K(X : Y\Z) 


1 - I<(X : Y, Z) 
1 - K(X : Z) 


(48) 


H(X\Y,Z) _ H(X\Y , Z) 
H(X\Z) H(X\Z) 


H(X\Z) - H(X\Y,Z) 
H(X\Z) 


1 _ H(X)-H(X\Y,Z) 

_ H(X) 

1 _ H(X)-H(Y\Z) 
H{X) 


1 - K{X : Y\Z) 


1 - K(X : Y, Z) 
1 - K(X : Z) 


Theorem 8. 


K{X : Y\Z) = 0 if and only if K{X :(Y\Z)) = 0 (49) 

Proof: This is obvious from Definition 4 and Definition 5. 
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Theorem 8 shows that the independence between X and Y, with the condition of Z for both 
X and Y, is equivalent to the independence between X and Y, with the condition of Z only for Y. 

Theorem 9. 


K(X :Y u Y 2 ,...,Y n ) = 


(50) 


K(X : Ti) + K(X : (y 2 |Pi)) + K(X : (T 3 |Ti, ^ 2 )) + • • • + K(X : (Y n \Y u YU)). 


Proof: 


K(X :Y 1 ,Y 2 ,...,Y n ) 


g(x)-g(x|yi,r 2 ,...,y n ) 

H(X) 


= —U- \(H(X) - H(X |Ti)) + (H(X |Ti) - H(X\Y U Y 2 )) + ■■■ + 
H(X) 


(H(X \Y U ..., Y n _ 1 ) - H(X \Y U Y n _ 1 , y n ))] 


= K(X : fi) + K(X : (y 2 |Ti)) + K{X : (y 3 |^i, * 2 )) + ■ ■ ■ + K(X : (Y n \Y U T n _i)). 

In (50), the total degree of X’s dependence on Y\,... ,Y n is actually the summation of the 
degrees of the X’s dependencies on each Yf s unique contribution to X, with common contribution 
removed. 

Theorem 10. 


K(X : Y) = K(Y : X) if and only if H(X) = H(Y), and (51) 

K(X : Y\Z) = K(Y : X\Z) if and only if H(X\Z) = H(Y\Z). (52) 

Proof: (51) is obvious from Definition 3 and the following the well-known equation for mutual 
information: 


H(X) - H(X\Y) = H(Y) - H(Y\X). 


(53) 
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H(X) — H(X\Y) in (53) is called mutual information among X and Y and denoted by I(X, Y). 

Equation (53) is also true if one substitutes X and Y with X\Z and Y\Z, respectively. Here, 
X\Z and Y\Z are the random vectors defined from random vectors X and Y, given condition Z. 
Thus, 


H(X\Z) - H(X\Y , Z) = H(Y\Z) - H(Y\X, Z). (54) 

Equation (54) plus Definition 4 are the proof of (52). 

Theorem 10 shows that, in general, K(X : Y) may or may not be the same as K(Y : X). 
This is called asymmetry. This property of asymmetry does not hold with linear correlations 
because linear relationships are always symmetric. The K-dependence coefficient measures the 
degree to which a group of random variables depend on another group of random variables. This 
dependence may or may not be linear. Generally speaking, for nonlinear dependence, one should 
not expect the measure to be always symmetric. 

4. Discussion 

This paper has defined the K-dependence coefficient and provided the proofs of some 
properties. The idea behind the K-dependence coefficient in (9) is that it represents the 
conditional entropy normalized by the unconditional entropy. Actually, other versions of this 
measure have appeared in the statistical literature, which have been discussed in the introduction. 
In 1988, Burg and Lewis compared two kinds of measures of multivariate association, based on 
Wilks’A and the Bartlett-Nanda-Pillai trace criterion V in terms of properties of the univariate 
R 2 . A unified set of derivations of the properties of R 2 is provided. These properties are also 
thought to be true for the K-dependence coefficient. Following the definition of the K-dependence 
coefficient, the partial K-dependence coefficient and the semipartial K-dependence coefficient are 
also defined (Charles Lewis, personnel communication, May 9, 2006). A set of the properties for 
(partial or semipartial) K-dependence coefficients is derived accordingly in Section 3. 

For the K-dependence coefficient, there are at least two aspects that are different from linear 
correlation. First, the K-dependence coefficient measures the degree to which a group of random 
variables depend on another group of random variables. This dependency may or may not be 
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linear. In fact, the dependence measured by the K-dependence coefficient includes both linear 
and nonlinear dependence among random variables. Secondly, the K-dependence coefficient may 
or may not be symmetric; that is, K(X,Y ) may or may not be the same as K(Y,X). Linear 
dependence is always symmetric; but for nonlinear dependence, symmetry does not hold if and 
only if H(X) ^ H(Y), according to Theorem 10. Generally speaking, symmetry is always true for 
linear dependence, but asymmetry in general holds for nonlinear dependence. 

Although the partial K-dependence coefficient and the senripartial K-dependence coefficients 
are defined separately, essentially, the senripartial K-dependence coefficient is a more general 
concept; that is, the partial K-dependence coefficient is a special case of the senripartial 
K-dependence coefficient. This is because 


K(X : Y\Z) = K((X\Z) : (Y\Z)). (55) 

The right-hand side of (55) is actually defined from the definition of the senripartial 
K-dependence coefficient. If Z is independent from X and Y, the left-hand side of (55) 
becomes the K-dependence coefficient. Therefore, the K-dependence coefficient and the partial 
K-dependence coefficient can be thought to be special cases of the senripartial K-dependence 
coefficient; that is, the K-dependence coefficient, the partial K-dependence coefficient, and the 
senripartial K-dependence coefficient actually are unified under the concept of the senripartial 
K-dependence coefficient. Therefore, any conclusions for the senripartial K-dependence coefficient 
are also relevant for the K-dependence coefficient and the partial K-dependence coefficient. 
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